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DETAILED ACTION 
Drawings 

1 . The drawings are objected to because in figure 8, parts 231 , 241 , 251 , 261 , 271 , 
and 281: should read "MPEG" and not "MPEC". Corrected drawing sheets in 
compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid 
abandonment of the application. Any amended replacement drawing sheet should 
include all of the figures appearing on the immediate prior version of the sheet, even if 
only one figure is being amended. The figure or figure number of an amended drawing 
should not be labeled as "amended." If a drawing figure is to be canceled, the 
appropriate figure must be removed from the replacement sheet, and where necessary, 
the remaining figures must be renumbered and appropriate changes made to the brief 
description of the several views of the drawings for consistency. Additional replacement 
sheets may be necessary to show the renumbering of the remaining figures. Each 
drawing sheet submitted after the filing date of an application must be labeled in the top 
margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If 
the changes are not accepted by the examiner, the applicant will be notified and 
informed of any required corrective action in the next Office action. The objection to the 
drawings will not be held in abeyance. 

Specification 

2. This application does not contain an abstract of the disclosure as required by 37 
CFR 1.72(b). An abstract on a separate sheet is required. 

The disclosure is objected to because of the following informalities: 
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On page 5, line 28: the word "on" should be replaced with "one". 
On page 38, line 22: part 408 is not shown in figure 13. 
Appropriate correction is required. 

Claim Objections 

3. Claims 2 and 26 objected to because of the following informalities: The phrase 
"and extraction processing" is not necessary as this is already stated in the claim. 
Appropriate correction is required. 

Claim 24 is objected to because of the following informalities: The phrase "said 
video object processing" lacks antecedent basis. Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1, 3-21, 23-25, 27-46, 48-67, and 69 are rejected under 35 U.S.C. 103(a) 

as being unpatentable over Palmer in view of MPEG-7 Requirements. 

4. Referring to claims 1 , Palmer discloses a system for generating a description 
record from video information (column 1, lines 10-12), comprising: at least one video 
input interface for receiving said video information (figure 2, part 36); a computer 
processor coupled to said at least one video input interface for receiving said video 
information therefrom (figure 2, part 30), processing said video information by 
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performing video object extraction processing to generate video object descriptions from 
said video information (column 1, lines 10-15), and processing said generated video 
object descriptions by entity relation graph generation processing to generate entity 
relation graph descriptions (column 7, lines 46-50); and a data storage system, 
operatively coupled to said processor for storing said at least one description record 
(column 4, lines 29-31). 

Palmer does not disclose a system where processing said generated video 
object descriptions by object hierarchy construction and extraction processing to 
generate video object hierarchy descriptions, and wherein at least one description 
record including said video object descriptions, said video object hierarchy descriptions 
and said entity relation graph descriptions is generated to represent content embedded 
within said video information. 

MEPG-7 Requirements disclose a system where processing said generated 
video object descriptions by object hierarchy construction and extraction processing to 
generate video object hierarchy descriptions (page 5, figure 1; page 10, lines 4-6), and 
wherein at least one description record including said video object descriptions, said 
video object hierarchy descriptions and said entity relation graph descriptions is 
generated to represent content embedded within said video information (page6, figure 
3). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to use a hierarchy of descriptions for the video descriptors disclosed in Palmer 
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as taught by MPEG-7 Requirements. The motivation for doing this would have been to 
allow for queries to be processed more efficiently (page 9, bullet 7). 

5. Referring to claim 3, Palmer discloses a system of claim 1, wherein said video 
object extraction processing comprises: video segmentation processing to segment 
each video in said video information into regions within said video (figure 4, part 67); 
and feature extraction and annotation processing to generate one or more feature 
descriptions for one or more said regions (figure 4, part 73); whereby said generated 
video object descriptions comprise said one or more feature descriptions for one or 
more said regions (figure 4 t part 76). 

Referring to claim 4, Palmer discloses a system of claim 3, wherein said regions 
are selected from the group consisting of local (figure 4, part 68), segment (figure 4, part 
65) and global regions (column 6, lines 63-66). 

6. Referring to claim 5, Palmer discloses a system of claim 3, wherein said one or 
more feature descriptions are selected from the group consisting of media features 
(column 5, lines 7-8), visual features (column 4, lines 56-57), and temporal features 
(column 4, line 55). 

Palmer does not disclose a system wherein one of the feature descriptions 
includes semantic features. 

MPEG-7 Requirements discloses a system wherein one of the feature 
descriptions includes semantic features (page 11, lines 28-29). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic features to the descriptor disclosed in Palmer as taught by 
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MPEG-7 Requirements. The motivation for doing this would have been to allow for 
more effective search of video files by having more descriptors describing the video 
(page 2, bullet 2, lines 4-6). 

7. Referring to claim 6, Palmer does not disclose a system of claim 5, wherein said 
semantic features are further defined by at least one feature description selected from 
the group consisting of who, what object, what action, where, when, why, and text 
annotation. 

MPEG-7 Requirements discloses a system of claim 5, wherein said semantic 
features are further defined by at least one feature description selected from the group 
consisting of who, what object, what action, where, when, why, and text annotation 
(page 11, lines 28-29). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic features to the descriptor disclosed in Palmer as taught by 
MPEG-7 Requirements. The motivation for doing this would have been to allow for 
more effective search of video files by having more descriptors describing the video 
(page 2, bullet 2, lines 4-6). 

8. Referring to claim 7, Palmer discloses a system of claim 5, wherein said visual 
features are further defined by at least one feature description selected from the group 
consisting of color, texture, position, size, shape, motion, camera motion, editing effect, 
and orientation (column 4, lines 56-57). 

Referring to claim 8, Palmer does not disclose a system of claim 5, wherein said 
media features are further defined by at least one feature description selected from the 
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group consisting of file format, file size, color representation, resolution, data file 
location, author, creation, scalable layer and modality transcoding. 

MPEG-7 Requirements discloses a system of claim 5, wherein said media 
features are further defined by at least one feature description selected from the group 
consisting of file format, file size, color representation, resolution, data file location, 
author, creation, scalable layer and modality transcoding (page 4, lines 23-31; page 4, 
bullet 2, lines 2-4). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add media features to the descriptor disclosed in Palmer as taught by MPEG- 
7 Requirements. The motivation for doing this would have been to allow for more 
effective search of video files by having more descriptors describing the video (page 2, 
bullet 2, lines 4-6). 

9. Referring to claim 9, Palmer discloses a system of claim 5, wherein said temporal 
features are further defined by at least one feature description selected from the group 
consisting of start time, end time and duration (column 4, lines 59-61). 

Referring to claim 10, Palmer does not disclose a system of claim 1, wherein said 
object hierarchy construction and extraction processing generates video object 
hierarchy descriptions of said video object descriptions based on visual feature 
relationships of video objects represented by said video object descriptions. 

MPEG-7 Requirements discloses a system of claim 1, wherein said object 
hierarchy construction and extraction processing generates video object hierarchy 
descriptions of said video object descriptions based on visual feature relationships of 
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video objects represented by said video object descriptions (page 9, bullet 7; page 4, 
lines 23-31). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add visual features to the descriptor disclosed in Palmer as taught by MPEG-7 
Requirements. The motivation for doing this would have been to allow for more 
effective search of video files by having more descriptors describing the video (page 2, 
bullet 2, lines 4-6). 

1 0. Referring to claim 1 1 , Palmer does not disclose a system of claim 1 , wherein said 
object hierarchy construction and extraction processing generates video object 
hierarchy descriptions of said video object descriptions based on semantic feature 
relationships of video objects represented by said video object descriptions. 

MPEG-7 Requirements discloses a system of claim 1, wherein said object 
hierarchy construction and extraction processing generates video object hierarchy 
descriptions of said video object descriptions based on semantic feature relationships of 
video objects represented by said video object descriptions (page 9, bullet 7; page 11, 
lines 28-29). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic features to the descriptor disclosed in Palmer as taught by 
MPEG-7 Requirements. The motivation for doing this would have been to allow for 
more effective search of video files by having more descriptors describing the video 
(page 2, bullet 2, lines 4-6). 
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Referring to claim 12, Palmer does not disclose a system of claim 1, wherein said 
object hierarchy construction and extraction processing generates video object 
hierarchy descriptions of said video object descriptions based on media feature 
relationships of video objects represented by said video object descriptions. 

MPEG-7 Requirements discloses a system of claim 1, wherein said object 
hierarchy construction and extraction processing generates video object hierarchy 
descriptions of said video object descriptions based on media feature relationships of 
video objects represented by said video object descriptions (page 11, lines 15-22). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add media features to the descriptor disclosed in Palmer as taught by MPEG- 
7 Requirements. The motivation for doing this would have been to allow for more 
effective search of video files by having more descriptors describing the video (page 2, 
bullet 2, lines 4-5). 

1 1 . Claim 1 3 is rejected based on the rejections of claims 1 0-1 2. 

Referring to claim 14, Palmer does not disclose a system of claim 1, wherein said 
object hierarchy construction and extraction processing generates video object 
hierarchy descriptions of said video object descriptions based on relationships of video 
objects represented by said video object descriptions, wherein said video object 
hierarchy descriptions have a plurality of hierarchical levels. 

MPEG-7 Requirements discloses a system of claim 1, wherein said object 
hierarchy construction and extraction processing generates video object hierarchy 
descriptions of said video object descriptions based on relationships of video objects 
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represented by said video object descriptions, wherein said video object hierarchy 
descriptions have a plurality of hierarchical levels (page 10, bullet 9). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to use a hierarchy of descriptions for the video descriptors disclosed in Palmer 
as taught by MPEG-7 Requirements. The motivation for doing this would have been to 
allow for queries to be processed more efficiently (page 9, bullet 7). 
12. Referring to claims 15-16, Palmer does not disclose a system of claim 14, 
wherein said video object hierarchy descriptions having a plurality of hierarchical levels 
comprise clustering hierarchies; wherein said clustering hierarchies are based on 
relationships of video objects represented by said video object descriptions, wherein 
said relationships are selected from a group consisting of visual feature relationships, 
semantic feature relationships, temporal relationships and media feature relationships. 

MPEG-7 Requirements discloses a system of claim 14, wherein said video object 
hierarchy descriptions having a plurality of hierarchical levels comprise clustering 
hierarchies (page 5, figure 1); wherein said clustering hierarchies are based on 
relationships of video objects represented by said video object descriptions, wherein 
said relationships are selected from a group consisting of visual feature relationships, 
temporal relationships, media feature relationships (see claim 13's rejection), and 
semantic feature relationships (page 11, lines 28-29). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic/visual/temporal/media features to the descriptor disclosed in 
Palmer as taught by MPEG-7 Requirements. The motivation for doing this would have 



Application/Control Number: 09/831 ,218 Page 1 1 

Art Unit: 2617 

been to allow for more effective search of video files by having more descriptors 
describing the video (page 2, bullet 2, lines 4-6). 

1 3. Referring to claims 1 7-18, Palmer does not disclose a system of claim 15, 
wherein said video object hierarchy descriptions having a plurality of hierarchical levels 
are configured to comprise multiple levels of abstraction hierarchies; wherein said 
multiple levels of abstraction hierarchies are configured to be based on relationships of 
video objects represented by said video object descriptions, wherein said relationships 
are selected from a group consisting of visual feature relationships, semantic feature 
relationships, temporal feature relationships and media feature relationships. 

MPEG-7 Requirements discloses a system of claim 15, wherein said video object 
hierarchy descriptions having a plurality of hierarchical levels are configured to comprise 
multiple levels of abstraction hierarchies (page 5, figure 1); wherein said multiple levels 
of abstraction hierarchies are configured to be based on relationships of video objects 
represented by said video object descriptions, wherein said relationships are selected 
from a group consisting of visual feature relationships, semantic feature relationships, 
temporal feature relationships and media feature relationships (see claim 15 & 16's 
rejection). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic/visual/temporal/media features to the descriptor disclosed in 
Palmer as taught by MPEG-7 Requirements. The motivation for doing this would have 
been to allow for more effective search of video files by having more descriptors 
describing the video (page 2, bullet 2, lines 4-6). 
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14. Referring to claim 19, Palmer discloses a system of claim 1 , wherein said entity 
relation graph generation processing generates entity relation graph descriptions of said 
video object descriptions based on relationships of video objects represented by said 
video object descriptions (column 7, lines 46-50), wherein said relationships are 
selected from the group consisting of visual feature relationships, temporal feature 
relationships and media feature relationships (column 4, lines 54-55). 

Palmer does not disclose a system wherein relationships are semantic feature 
relationships. 

MPEG-7 Requirements discloses a system wherein relationships are semantic 
feature relationships (page 11, lines 28-29). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic features to the descriptor disclosed in Palmer as taught by 
MPEG-7 Requirements. The motivation for doing this would have been to allow for 
more effective search of video files by having more descriptors describing the video 
(page 2, bullet 2, lines 4-6). 

Referring to claim 20, Palmer discloses a system of claim 1 , further comprising a 
data storage system is operative to store said encoded description information as said 
at least one description record (column 4, lines 29-31). 

Palmer does not disclose an encoder for receiving and encoding said video 
object descriptions into encoded description information. 

MPEG-7 Requirements discloses an encoder for receiving and encoding said 
video object descriptions into encoded description information (page 6, figure 3). 
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At the time of the invention it would have been obvious to one of ordinary skill in 
the art to use an encoder as taught by MPEG-7 Requirements in the video descriptor 
disclosed in Palmer. The motivation for doing this would have been to encapsulate the 
data for searching into the video file itself enabling simpler transferring over the internet. 
1 5. Referring to claim 21 , Palmer discloses a system of claim 1 , wherein said video 
object descriptions (column 1, lines 10-12) and said entity relation graph descriptions 
are created (column 7, lines 46-50); and wherein said data storage system is operative 
to store said encoded description information as said at least one description record 
(column 4, lines 29-31). 

Palmer does not disclose a system where said video object hierarchy 
descriptions and the other descriptions are combined together to form video 
descriptions, and further comprising an encoder for receiving and encoding said video 
descriptions into encoded description information. 

MPEG-7 Requirements discloses a system where said video object hierarchy 
descriptions and the other descriptions are combined together to form video 
descriptions, and further comprising an encoder for receiving and encoding said video 
descriptions into encoded description information (page 5, figure 1; page 6, figure 3). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to use an encoder as taught by MPEG-7 Requirements in the video descriptor 
disclosed in Palmer. The motivation for doing this would have been to encapsulate the 
data for searching into the video file itself enabling simpler transferring over the internet. 
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1 6. Referring to claim 23, Palmer does not disclose a system of claim 21 , wherein 
said encoder comprises an XML encoder. 

MPEG-7 Requirements discloses a system of claim 21, wherein said encoder 
comprises an XML encoder (page 3, line 21). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to encoded the descriptions in an XML format as taught by MPEG-7 
Requirements in the video description system disclosed by Palmer. The motivation for 
doing this would have been to use an existing standard that is widely known in the 
industry. 

Referring to claim 24, Palmer discloses a system of claim 1 , further comprising: a 
video display device operatively coupled to the computer processor for displaying the 
video information (figure 2, part 34); and at least one user input device operatively 
coupled to the computer processor (figure 2, parts 16 and 17), wherein at least a portion 
of said video object processing includes receiving a user input through manipulation of 
said user input device (column 2, line 23; Note: a manual mode is being interpreted as 
equivalent to a device requiring user input). 

17. Referring to claim 37, Palmer does not disclose a method of claim 25, wherein 
said step of object hierarchy construction and extraction processing generates video 
object hierarchy descriptions of said video object descriptions based on temporal 
feature relationships of video objects represented by said video object descriptions. 

MPEG-7 Requirements discloses a method of claim 25, wherein said step of 
object hierarchy construction and extraction processing generates video object 
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hierarchy descriptions of said video object descriptions based on temporal feature 
relationships of video objects represented by said video object descriptions (page 8, 
bullet 12b). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add temporal features to the descriptor disclosed in Palmer as taught by 
MPEG-7 Requirements. The motivation for doing this would have been to allow for 
more effective search of video files by having more descriptors describing the video 
(page 2, bullet 2, lines 4-6). 

1 8. Referring to claim 43, Palmer does not disclose a method of claim 40, wherein 
said multiple levels of abstraction hierarchies are configured to be based on 
relationships of video objects represented by said video object descriptions, wherein 
said relationships are selected from a group consisting of visual feature relationships, 
semantic feature relationships, temporal feature relationships and media feature 
relationships. 

MPEG-7 Requirements discloses a method of claim 40, wherein said multiple 
levels of abstraction hierarchies are configured to be based on relationships of video 
objects represented by said video object descriptions (page 9, bullet 7), wherein said 
relationships are selected from a group consisting of visual feature relationships, 
semantic feature relationships, temporal feature relationships and media feature 
relationships (page 6-7, table 1; page 8, bullet 12). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic/visual/temporal/media features to the descriptor disclosed in 
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Palmer as taught by MPEG-7 Requirements. The motivation for doing this would have 
been to allow for more effective search of video files by having more descriptors 
describing the video (page 2, bullet 2, lines 4-6). 

19. Referring to claims 63 and 64, Palmer does not disclose a computer readable 
media of claim 62, wherein said video object hierarchy descriptions having a plurality of 
hierarchical levels are configured to comprise multiple levels of abstraction hierarchies; 
wherein said multiple levels of abstraction hierarchies are configured to be based on 
relationships of video objects represented by said video object descriptions, wherein 
said relationships are selected from a group consisting of visual feature relationships, 
semantic feature relationships, temporal feature relationships and media feature 
relationships. 

MPEG-7 Requirements disclose a computer readable media of claim 62, wherein 
said video object hierarchy descriptions having a plurality of hierarchical levels are 
configured to comprise multiple levels of abstraction hierarchies (page 5, figure 1); 
wherein said multiple levels of abstraction hierarchies are configured to be based on 
relationships of video objects represented by said video object descriptions, wherein 
said relationships are selected from a group consisting of visual feature relationships, 
semantic feature relationships, temporal feature relationships and media feature 
relationships (page 6-7, table 1; page 8, bullet 12). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to use a hierarchy of descriptions for the video descriptors disclosed in Palmer 
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as taught by MPEG-7 Requirements. The motivation for doing this would have been to 
allow for queries to be processed more efficiently (page 9, bullet 7). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to add semantic/visual/temporal/media features to the descriptor disclosed in 
Palmer as taught by MPEG-7 Requirements. The motivation for doing this would have 
been to allow for more effective search of video files by having more descriptors 
describing the video (page 2, bullet 2, lines 4-6). 
20. Claims 25 and 49 are rejected on the same grounds as claim 1 . 

Claims 27 and 50 are rejected on the same grounds as claim 3. 

Claim 28 is rejected on the same grounds as claim 4. 

Claims 29 and 51 are rejected on the same grounds as claim 5. 

Claims 30 and 52 are rejected on the same grounds as claim 6. 

Claims 31 and 53 are rejected on the same grounds as claim 7. 

Claims 32 and 54 are rejected on the same grounds as claim 8. 

Claims 33 and 55 are rejected on the same grounds as claim 9. 

Claims 34 and 56 are rejected on the same grounds as claim 10. 

Claims 35 and 57 are rejected on the same grounds as claim 11. 

Claims 36 and 58 are rejected on the same grounds as claim 12. 

Claim 38 is rejected on the same grounds as claim 13. 

Claims 39 and 60 are rejected on the same grounds as claim 14. 

Claims 40 and 61 are rejected on the same grounds as claim 15. 

Claims 41 and 62 are rejected on the same grounds as claim 16. 



Application/Control Number: 09/831,218 Page 18 

Art Unit: 2617 

Claim 42 is rejected on the same grounds as claim 17. 
Claims 44 and 65 are rejected on the same grounds as claim 19. 
Claims 45 and 66 are rejected on the same grounds as claim 20. 
Claims 46 and 67 are rejected on the same grounds as claim 21 . 
Claims 48 and 69 are rejected on the same grounds as claim 23. 
Claim 59 is rejected on the same grounds as claim 37. 

21. Claims 22, 47, 68, and 70-75 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Palmer in view of MPEG-7 Requirements as applied to claims 1, 21, 
25, 46, 49, 67 above, and further in view of Mills. 

Referring to claim 22, Palmer and MPEG-7 Requirements do not disclose a 
system of claim 21, wherein said encoder comprises a binary encoder. 

Mills discloses a system of claim 21 , wherein said encoder comprises a binary 
encoder (column 20, lines 61-65). 

At the time of the invention it would have been obvious to a person of ordinary 
skill in the art to use the binary encoding as taught by Mills in the system disclosed by 
Palmer and MPEG-7 Requirements. The motivation for doing this would have been to 
save space in the file by encoding common terms to that require less bits and therefore 
saves space (column 20, lines 65-66). 

22. Claims 47 and 68 are rejected on the same grounds as 22. 

Referring to claims 70-75, MPEG-7 Requirements discloses a system where the 
descriptions are writing in XML (page 3, line 21). 
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Palmer and MPEG-7 Requirements do not disclose a system where the feature 
descriptions include pointers to extraction and matching code to facilitate code 
downloading. 

Mills discloses a system where the feature descriptions include pointers to 
extraction and matching code to facilitate code downloading (column 1, lines 29-37; 
Note: XML code in websites, which facilitate the downloading of files, is being 
interpreted as being equivalent to including pointers to facilitate downloading). 

At the time of the invention it would have been obvious to one of ordinary skill in 
the art to encoded the descriptions in an XML format as taught by MPEG-7 
Requirements in the video description system disclosed by Palmer. The motivation for 
doing this would have been to use an existing standard that is widely known in the 
industry. 

Allowable Subject Matter 

23. Claims 2 and 26 are objected to as being dependent upon a rejected base claim, 
but would be allowable if rewritten in independent form including all of the limitations of 
the base claim and any intervening claims. 

Conclusion 

24. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

MPEG-7 Context and Objectives; October 98. 
MPEG-7 Proposal Package Description; October 98. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Justin E. Shepard whose telephone number is (571) 
272-5967. The examiner can normally be reached on 7:30-5 M-F. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Chris Kelley can be reached on (571) 272-7331. The fax phone number for 
the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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